improve encoding handling for `setup.cfg` #1180

benoit-pierre · 2017-10-25T19:43:01Z

Support the same mechanism as for Python sources for declaring the encoding to be used when reading setup.cfg (see PEP 263), and return the results of reading it as Unicode.

Fix #1062 and #1136.

Support the same mechanism as for Python sources for declaring the encoding to be used when reading `setup.cfg` (see PEP 263), and return the results of reading it as Unicode. Fix pypa#1062 and pypa#1136.

jaraco

Does it make sense to defer this work until setuptools adopts distutils (#417)?

setuptools/py36compat.py

jaraco · 2017-10-25T21:14:43Z

setuptools/py36compat.py

@@ -33,11 +47,16 @@ def parse_config_files(self, filenames=None):
        if DEBUG:
            self.announce("Distribution.parse_config_files():")

-        parser = ConfigParser(interpolation=None)
+        parser = ConfigParser()


I know interpolation causes problems when a percent appears in the setup.cfg (such as for a password). I think there are tests to protect this behavior (perhaps only in CPython).

It's the other way round, interpolation as always been enabled, see: #889 (comment)

This compatibility code was actually not used, that's why I added a test first to make sure this behavior (interpolation enabled) did not change.

Oh, right. Good point. I wonder then if that means we (I) need to update the code in distutils for Python 3.7 to restore interpolation before it goes live.

Okay, reviewing the history on the CPython issue, I see that there's not yet a patch in CPython that's affected by this change.

benoit-pierre · 2017-10-25T21:24:49Z

Reworked and slightly expanded tests.

benoit-pierre · 2017-10-25T21:28:58Z

I agree that this could wait for distutils code to be included in setuptools.

jaraco · 2017-11-20T02:15:20Z

We can either wait for the distutils code to be adopted by Setuptools or we can simply override the parse_config_files in Distribution (and own it, rather than working to rely on the upstream version). I'm slightly inclined toward the latter, not knowing when/if distutils will land. Would you like to work on those changes or would you rather I do it (I feel some responsibility having put the compatibility layer in place)?

RonnyPfannschmidt · 2018-06-21T08:12:34Z

ouch, this is still an issue :/

c00kiemon5ter · 2018-07-03T01:18:13Z

Hi and thanks for working on this. I also got bitten by #1136 and followed the thread here. Is this planned to be merged?

A bug is blocking setuptools from working with python2 [bug]. Work is on its way [pr]. Until that is fixed, package_dir should be defined in setup.py to preserve compatibility of the native str type. [bug]: pypa/setuptools#1136 [pr]: pypa/setuptools#1180 Signed-off-by: Ivan Kanakarakis <ivan.kanak@gmail.com>

jaraco · 2018-07-05T00:00:01Z

I think we do basically want to adopt this change, but as part of a refactor to have setuptools take ownership of the relevant functionality from distutils (rather than trying to contribute it upstream first, which was the intention here).

jaraco · 2019-01-25T17:33:34Z

I'm sorry to have left this to languish for so long. I want to accept these changes with some small tweaks (mainly proper adoption of the config file parsing functionality). I'll resolve the conflicts and press forward.

…e upstream, just incorporate the functionality directly.

jaraco · 2019-01-25T21:53:51Z

I believe this work is ready to go, but tests are failing due to #1644.

jaraco · 2019-01-27T00:57:35Z

There's still a failing test on Python 2.7. I'm not yet clear on why.

benoit-pierre · 2019-01-27T01:54:21Z

That's an issue somewhere in the bowels of distutils, when handling boolean options:

 setuptools/dist.py | 45 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git i/setuptools/dist.py w/setuptools/dist.py
index 4cc3bdfe..6cabfb60 100644
--- i/setuptools/dist.py
+++ w/setuptools/dist.py
@@ -12,6 +12,7 @@
 import distutils.cmd
 import distutils.dist
 from distutils.errors import DistutilsOptionError
+from distutils.fancy_getopt import translate_longopt
 from distutils.util import strtobool
 from distutils.debug import DEBUG
 import itertools
@@ -560,6 +561,50 @@ def _clean_req(self, req):
         req.marker = None
         return req
 
+    def _set_command_options(self, command_obj, option_dict=None):
+        """Set the options for 'command_obj' from 'option_dict'.  Basically
+        this means copying elements of a dictionary ('option_dict') to
+        attributes of an instance ('command').
+
+        'command_obj' must be a Command instance.  If 'option_dict' is not
+        supplied, uses the standard option dictionary for this command
+        (from 'self.command_options').
+        """
+        command_name = command_obj.get_command_name()
+        if option_dict is None:
+            option_dict = self.get_option_dict(command_name)
+
+        if DEBUG:
+            self.announce("  setting options for '%s' command:" % command_name)
+        for (option, (source, value)) in option_dict.items():
+            if DEBUG:
+                self.announce("    %s = %s (from %s)" % (option, value,
+                                                         source))
+            try:
+                bool_opts = [translate_longopt(o)
+                             for o in command_obj.boolean_options]
+            except AttributeError:
+                bool_opts = []
+            try:
+                neg_opt = command_obj.negative_opt
+            except AttributeError:
+                neg_opt = {}
+
+            try:
+                is_string = isinstance(value, six.string_types)
+                if option in neg_opt and is_string:
+                    setattr(command_obj, neg_opt[option], not strtobool(value))
+                elif option in bool_opts and is_string:
+                    setattr(command_obj, option, strtobool(value))
+                elif hasattr(command_obj, option):
+                    setattr(command_obj, option, value)
+                else:
+                    raise DistutilsOptionError(
+                        "error in %s: command '%s' has no such option '%s'"
+                        % (source, command_name, option))
+            except ValueError as msg:
+                raise DistutilsOptionError(msg)
+
     def _parse_config_files(self, filenames=None):
         """
         Adapted from distutils.dist.Distribution.parse_config_files,

jaraco · 2019-01-27T02:34:36Z

The remaining test failure (just one) is due to an option "tag-date=0" being parsed as u'0' but on Python 2 not being interpreted as isinstance(tag_date, str) (ref).

…ter string detection.

…onfigParser.

jaraco · 2019-01-27T02:50:17Z

@benoit-pierre Assuming tests pass (they do for me locally), would you please review the latest changes?

benoit-pierre · 2019-01-27T12:37:26Z

Well, I can't approve my own PR, but looks good to me!

PR pypa#1180 and 24be5ab changed the way we read setup.cfg, but when editing setup.cfg we might still lose the encoding hint on the file, which can cause failures in subsequent steps. This change detectes the encoding like in `_parse_config_files` and writes an encoding hint header if the file encoding was detected. Fixes jaraco/configparser#37.

See pypa/setuptools#1180.

benoit-pierre added 2 commits October 25, 2017 17:50

add test to ensure setup.cfg interpolation behavior remain unchanged

3686ded

improve encoding handling for setup.cfg

2c897b5

Support the same mechanism as for Python sources for declaring the encoding to be used when reading `setup.cfg` (see PEP 263), and return the results of reading it as Unicode. Fix pypa#1062 and pypa#1136.

jaraco requested changes Oct 25, 2017

View reviewed changes

benoit-pierre force-pushed the fix_889_and_non-ascii_in_setup.cfg_take_2 branch from e110db5 to 2c897b5 Compare October 25, 2017 21:17

benoit-pierre added the draft label Dec 22, 2017

hectormartinezdev mentioned this pull request Jan 24, 2018

package_dir in setup.cfg #1259

Closed

jaraco added 2 commits January 25, 2019 15:49

Merge branch 'master' into fix_889_and_non-ascii_in_setup.cfg_take_2

992aa3d

Given that the config file parsing functionality is unlikely to chang…

24be5ab

…e upstream, just incorporate the functionality directly.

jaraco approved these changes Jan 25, 2019

View reviewed changes

Update changelog

9ef3ce6

jaraco closed this Jan 27, 2019

jaraco reopened this Jan 27, 2019

jaraco added 2 commits January 26, 2019 19:51

Merge branch 'master' into fix_889_and_non-ascii_in_setup.cfg_take_2

62e4c1d

Merge branch 'master' into fix_889_and_non-ascii_in_setup.cfg_take_2

80082fa

jaraco added 2 commits January 26, 2019 21:45

Adopt distutils.dist.Distribution._set_command_options to support bet…

b1615d1

…ter string detection.

Fix test failure by better detecting string options from an updated C…

249f24a

…onfigParser.

jaraco merged commit b31997d into pypa:master Jan 27, 2019

jaraco mentioned this pull request Jan 27, 2019

[BUG] Invalid string type when reading declarative setup.cfg under Python 2 #1136

Closed

benoit-pierre deleted the fix_889_and_non-ascii_in_setup.cfg_take_2 branch January 27, 2019 14:38

benoit-pierre removed the draft label Jan 28, 2019

This was referenced Jan 28, 2019

TypeError: 'output_dir' must be a string or None since setuptools 40.7.0 #1653

Closed

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-t_utKw/pyodbc/ #1654

Closed

Revert "improve encoding handling for setup.cfg" #1656

Closed

jaraco mentioned this pull request Feb 23, 2019

setup.cfg should standardize on UTF-8 for encoding #1702

Closed

itsadok mentioned this pull request Mar 30, 2019

Preserve encoding with edit_config #1729

Closed

Ge0 pushed a commit to Ge0/cookiecutter-python that referenced this pull request Jun 2, 2019

Remove setuptools workaround

582f8ed

See pypa/setuptools#1180.

jmbowman mentioned this pull request May 4, 2020

Ensure setup.cfg is only ASCII. CRI-169 openedx/edx-platform#23872

Merged

fireattack mentioned this pull request Sep 28, 2020

PIP doesn't read setup.cfg in UTF-8, which causes UnicodeDecodeError pypa/pip#8931

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve encoding handling for `setup.cfg` #1180

improve encoding handling for `setup.cfg` #1180

benoit-pierre commented Oct 25, 2017

jaraco left a comment

jaraco Oct 25, 2017

benoit-pierre Oct 25, 2017

jaraco Nov 20, 2017

jaraco Nov 20, 2017

benoit-pierre commented Oct 25, 2017

benoit-pierre commented Oct 25, 2017

jaraco commented Nov 20, 2017 •

edited

RonnyPfannschmidt commented Jun 21, 2018

c00kiemon5ter commented Jul 3, 2018

jaraco commented Jul 5, 2018

jaraco commented Jan 25, 2019

jaraco commented Jan 25, 2019

jaraco commented Jan 27, 2019

benoit-pierre commented Jan 27, 2019

jaraco commented Jan 27, 2019

jaraco commented Jan 27, 2019

benoit-pierre commented Jan 27, 2019

improve encoding handling for setup.cfg #1180

improve encoding handling for setup.cfg #1180

Conversation

benoit-pierre commented Oct 25, 2017

jaraco left a comment

Choose a reason for hiding this comment

jaraco Oct 25, 2017

Choose a reason for hiding this comment

benoit-pierre Oct 25, 2017

Choose a reason for hiding this comment

jaraco Nov 20, 2017

Choose a reason for hiding this comment

jaraco Nov 20, 2017

Choose a reason for hiding this comment

benoit-pierre commented Oct 25, 2017

benoit-pierre commented Oct 25, 2017

jaraco commented Nov 20, 2017 • edited

RonnyPfannschmidt commented Jun 21, 2018

c00kiemon5ter commented Jul 3, 2018

jaraco commented Jul 5, 2018

jaraco commented Jan 25, 2019

jaraco commented Jan 25, 2019

jaraco commented Jan 27, 2019

benoit-pierre commented Jan 27, 2019

jaraco commented Jan 27, 2019

jaraco commented Jan 27, 2019

benoit-pierre commented Jan 27, 2019

improve encoding handling for `setup.cfg` #1180

improve encoding handling for `setup.cfg` #1180

jaraco commented Nov 20, 2017 •

edited